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Program description 


Career Academies are school-within-school programs operating 
in high schools. They offer career- related curricula based on 


a career theme, academic coursework, and work experience 
through partnerships with local employers. 1 


Research 


One study of Career Academies met What Works Clearinghouse 
(WWC) evidence standards. This randomized controlled trial 
included 474 youth who were predicted to be most at-risk of 


dropping out of high school prior to the intervention. 2 The Acad- 
emies were located in eight urban areas in six states. 


Effectiveness 


Career Academies were found to have potentially positive 
effects on staying in school, potentially positive effects on 
progressing in school, and no discernible effects on completing 
school for those youth most at-risk of dropping out prior to the 


intervention. 3 The Career Academies served a more heteroge- 
neous population, and the results for the high-risk youth may 
not be independent of their participation in the intervention with 
youth less at risk of dropping out. 



Staying in school Progressing in school Completing school 

Rating of effectiveness Potentially positive effects Potentially positive effects No discernible effects 

Improvement index 4 Average-. +13 percentile points Average: +13 percentile points Average: -0.1 percentile 

Range: +11 to +15 percentile points points 

1. This report focuses on Career Academies with a school-within-school structure. Some Career Academies have operated as entire schools but are 
outside the scope of the review because their primary focus is not dropout prevention. 

2. This report focuses on the 474 youth in the study sample who were most at risk of dropping out of high school because the Career Academies model 
initially focused on high-risk youth; these youth represent 27% of the total study sample of 1,764. Researchers used student background characteristics 
(including sibling dropped out, overage for grade, transferred schools two or more times, and attendance, GPA, and credits earned in the year of ran- 
dom assignment) to develop a model to predict whether students in the comparison group dropped out of school, and then applied the estimated model 
to predict which intervention-group students were most likely to drop out. The findings for those youth considered less at-risk of dropping out of school 
are presented in Appendices A4.1-A4.3. 

3. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

4. These values show the average and range of improvement indices for all findings in the three review domains across the one study included in this 
report. The range is provided only if more than one outcome was measured within a domain. 
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Additional program 
information 



Research 



Effectiveness 
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Developer and contact 

Information on the history of Career Academies and current 
resources for program implementation is available from the 

National Career Academy Coalition (NCAC) and the Career 
Academy Support Network (CASNf . 

Scope of use 

The NCAC reports that at least 1,500 Career Academies are 
currently in operation, and a registry maintained by the CASN 
includes more than 1,600 Career Academies. 

Description of intervention 

Career Academies were developed more than 30 years ago as 
a drop-out prevention strategy and targeted youth considered 
most at risk of dropping out of high school. More recently, 
Career Academies have broadened the kinds of students they 
serve, consistent with efforts to integrate rigorous academic 
curricula with career themes and to attract students who are 



The WWC reviewed seven studies of the effectiveness of Career 
Academies. One study (Kemple & Snipes, 2000; Kemple, 2004) 
was a randomized controlled trial that met WWC evidence stan- 
dards. Six studies did not meet WWC evidence screens. 

The Kemple and Snipes (2000) and Kemple (2004) study was 
a randomized controlled trial that included a total of 1 ,764 



preparing for post-secondary education. Career Academies 
operate within a larger high school and are guided by a career 
theme such as health care, finance, technology, communica- 
tions, and public service. Students take their career-related 
courses within the Academy, which often are taught by the core 
team of Academy teachers. Some Academies integrate their 
courses with other academic subjects required for graduation. 
Career Academies also partner with local employers, who 
provide internship opportunities and mentoring to students, 
contribute resources, participate in special events, and serve on 
Academy advisory boards. 

Cost 

Information on the cost of Career Academies was found for the 
California Partnership Academies and was estimated in 2004 to 
be $600 a pupil more than a district’s average per pupil expern- 
diture. 5 The WWC did not find information on the cost to deliver 
services to the high-risk youth within the Career Academies. 



students who applied to the entrance grade (9th or 10th) of nine 
Career Academies included in the evaluation. Of these, 474 stu- 
dents were predicted to be at high risk of dropping out of high 
school. 2 The study measured outcomes at the end of a student’s 
projected 12th-grade year and then four years after a student’s 
projected 12th-grade year. 



Findings 2 

The WWC review of interventions for dropout prevention 
addresses student outcomes in three domains: staying in school, 
progressing in school, and completing school. 

Staying in school. Kemple and Snipes (2000) reported that for 
the sample of youth most at risk of dropping out of high school, 



Career Academies had a positive and statistically significant 
effect on dropping out. At the end of the students’ projected 
12th-grade year, 21% of the Career Academy group and 32% of 
the comparison group had dropped out of high school. Findings 
for youth who were predicted to have a low or medium risk of 
dropping out of high school are presented in Appendix A4.1. 



5. This estimate is derived from the following sources: www2.bc.cc.ca.us/techprep/partnershipplus.html and www.ncset.ora/publications/essentialtools/ 
dropout/part3.3.02.asp . 
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Progressing in school. Kemple and Snipes (2000) reported that 
for the sample of youth most at risk of dropping out of high 
school, Career Academies had a positive and statistically sig- 
nificant effect on progressing through high school. At the end of 
the students’ projected 12th-grade year, Career Academy youth 
had earned an average of 19 credits and comparison youth had 
earned an average of 17 credits, and 40% of Career Academy 
youth and 26% of comparison youth had earned sufficient cred- 
its to graduate. Findings for youth who were predicted to have a 
low or medium risk of dropping out of high school are presented 
in Appendix A4.2. 

Completing school. Kemple (2004) reported that four years after 
students’ projected 12th-grade year, there was no statistically 
significant difference between the percentage of high-risk 
Career Academy and comparison youth who earned a diploma 
or GED certificate; 83% of the youth in both groups had either 
graduated with a diploma or received a GED. Findings for youth 
who were predicted to have a low or medium risk of dropping 
out of high school are presented in Appendix A4.3. 

Rating of effectiveness 

The WWC rates an intervention’s effects for a given outcome 
as positive, potentially positive, mixed, no discernible effects, 
potentially negative, or negative. The rating of effectiveness 
takes into account four factors: the quality of the research 
design, the statistical significance of the findings, 6 the size of 
the difference between participants in the intervention condition 
and the comparison condition, and the consistency in findings 
across studies (see the WWC Intervention Rating Scheme) . 

The WWC found Career Academies to have potentially positive 
effects on staying in school, potentially positive effects on 



progressing in school, and no discernible effects on completing 
school. 

Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see Technical Details 
of WWC-Conducted Computations) . The improvement index rep- 
resents the difference between the percentile rank of the average 
student in the intervention condition versus the percentile rank 
of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is entirely based 
on the size of the effect, regardless of the statistical significance 
of the effect, the study design, or the analysis. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting favorable results for the intervention group. 
The improvement index for staying in school is +13 percentile 
points based on one study. The average improvement index 
for progressing in school is +13 percentile points based on one 
study, with a range of +11 to +15 percentile points across the 
findings. The improvement index for completing school is -0.1 
percentile point based on one study. 

Summary 

The WWC reviewed seven studies on Career Academies. One of 
these studies met WWC evidence standards, and the remaining 
six studies did not meet WWC evidence screens. Based on this 
one study, the WWC found potentially positive effects on staying 
in school, potentially positive effects on progressing in school, 
and no discernible effects on completing school. The evidence 
presented in this report is limited and may change as new 
research emerges. 



6. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 

classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Career Academies, a correction for multiple com- 
parisons was needed for the multiple measures reported in the progressing in school domain. 
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Appendix 



Appendix A1 


Study characteristics: Kemple & Snipes, 2000 (randomized controlled trial) and Kemple, 2004 (randomized controlled trial) 


Characteristic 


Description 


Study citation 


Kemple, J. J., & Snipes, J. C. (2000). Career Academies: Impacts on students’ engagement and performance in high school. New York: MDRC (Manpower Demonstration 
Research Corporation). 


Participants 


Additional Sources: 

Kemple: J. J. (2004). Career Academies: Impacts on labor market outcomes and educational attainment. New York: MDRC (Manpower Demonstration Research 
Corporation). 

This review focuses on the subgroup of 474 youth in the study sample who were considered most at risk of dropping out prior to the start of the intervention. These youth 
represent 27% of the total study sample of 1 ,7644 Among these high-risk youth, 79% were 15 years old or younger, a majority were female (57%), and many were Hispanic 
(52%) or African-American (38%), lived in a single-parent household (50%), and received welfare or Food Stamps (32%). In the year prior to random assignment, 33% of 
the high-risk youth were absent for at least 15% of the school year. In the year of random assignment, 62% of the high-risk youth earned a grade point average of 2.0 or lower 
and 43% were overage for their grade level. 


Setting 


The nine schools in the evaluation were in eight urban areas in six states: Pittsburgh, Pennsylvania; Baltimore, Maryland; Washington, DC; Miami Beach, Florida; Socorro, 
Texas; Santa Ana, California; Watsonville, California; and San Jose, California. 



(continued) 

1. Researchers used student background characteristics (including sibling dropped out, overage for grade, transferred schools two or more times, and attendance, GPA, and credits earned in 
year of random assignment) to develop a model to predict whether students in the comparison group dropped out of school, and then applied the estimated model to predict which intervention- 
group students were most likely to drop out. 
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Appendix A1 Study characteristics: Kemple & Snipes, 2000 (randomized controlled trial) and Kemple, 2004 (randomized controlled trial) 

(continued) 



Characteristic 


Description 


Intervention 2 


The intervention group was randomly assigned to the Career Academy to which they applied, and 86% of the high-risk youth randomly assigned to the Academy group 
enrolled in an Academy. The intervention group could attend the Academy until graduation. Career Academies in the evaluation had been operating several years before the 
study began. They had three primary components: a school-within-school organization with a career theme, academic plus vocational curricula related to the career theme, 
and employer partnerships. 

• School-within-school organization with a career theme. The Career Academies in the study were organized around six career themes: business and finance, electronics 
and aerospace technology, health, public service, travel and tourism, and video technology. Two Academies admitted students in ninth grade, and seven admitted stu- 
dents in 10th grade. Most Career Academies enrolled 50 to 75 students per grade (the average Academy class size was similar to class sizes in the host high schools). 

A group of two to nine teachers taught classes exclusively within the Academy, and students had the opportunity to have the same teacher for several years. In half of 
the sites, the Academy teachers had regularly scheduled common planning time. Some Academies were managed by a teacher who served as a liaison between the 
Academy and the school and district, while others were managed by a district administrator overseeing multiple Academies within the district. 

• Academic plus vocational curricula related to the career theme. Academies offered a sequence of career-related classes. Students took two to four courses each year in their 
Academy. The remaining courses, including core academic requirements for graduation, were usually taken in the host high school. In some Academies, academic and career- 
related courses were integrated. Researchers observed that the academic curricula and instructional practices were similar between the intervention and comparison conditions. 

• Employer partnerships. Formal relationships with employers in the community supported Academy programs and provided career-related activities for students. A range 
of 3-54 employer partners were associated with each Academy. All Academies provided internship opportunities to the students through the employer partners, and 
many of these took place in the summer between the 11th and 12th grade. Many Academies set minimum criteria, mostly related to academic progress, for students 
seeking internship positions. Additionally, employer partners contributed funds and other resources, assisted Academies in identifying relevant activities for students, 
participated as speakers and mentors, and served on advisory boards of some Academies. 


Comparison 


Youth not randomly offered admission to a Career Academy constituted the comparison group. Most comparison group youth enrolled in a general education program in the 
host high school. Some enrolled in citywide magnet programs or specialty schools. About 3% enrolled in a Career Academy that was in the evaluation, despite being in the 
comparison group. 


Primary outcomes 
and measurement 


Staying in school domain (one outcome from a student survey, district records, and school enrollment status reports): the percentage of students who dropped out of high 
school before the end of their projected 12th-grade year. Progressing in school domain (two outcomes from school records data): total course credits accumulated between 
9th grade and the end of the projected 12th-grade year, and the percentage of students whose accumulated credits met their school’s graduation requirements. Completing 
school domain (one outcome from a student survey administered four years after a student's projected 12th-grade year): earned a high school diploma or GED certificate. 
(See Appendices A2.1-A2.3 for more detailed descriptions of outcome measures.) 


Teacher characteristics 
and training 


Career Academy teachers came from a variety of academic and vocational disciplines but generally were similar to other teachers in host high schools. Some of the 
professional development opportunities offered to Academy teachers included learning how best to support students in a small learning environment and learning strategies 
for coordinating career development and employer-related activities. The smaller school-within-school structure lent itself to opportunities for shared planning time among 
Academy teachers. 



2. Details on the structural components of the Academies are found in Kemple & Rock (1996); this report does not contain data on student outcomes. 



WWC Intervention Report Career Academies 



October 5, 2006 



Appendix A2.1 Outcome measure for the staying in school domain 



Outcome measure 


Description 


Dropped out of high school 


This binary measure counted a student as a dropout at the end of their scheduled 12th-grade year if they were not listed as enrolled on any one of three data sources 
(student survey, district records, and school enrollment status reports completed by the host high schools) and if one of the following conditions were met: student reported 
being a dropout on the survey or school records indicated student had dropped out with no indication of being enrolled elsewhere (as cited in Kemple & Snipes, 2000 and 
through further author communication). 



Appendix A2.2 Outcome measure for the progressing in school domain 



Outcome measure 


Description 


Total course credits earned 


This continuous measure taken from school records data includes all course credits students earned from ninth grade through the end of their projected 12th-grade year 
(until just before they would have graduated from high school). In seven of the study sites, the Academies began in the 10th grade and ninth-grade course credits were 
earned prior to the intervention. The remaining two Academies began in the ninth grade and the course credits were earned during four years of exposure to the intervention 
(as cited in Kemple & Snipes, 2000). 


Credits earned met 
graduation requirements 


This binary measure taken from school records data indicates whether the credits earned from 9th grade through the end of their projected 12th-grade year (until just 
before they would have graduated from high school) met the requirements for graduation of each study school (as cited in Kemple & Snipes, 2000). 



Appendix A2.3 Outcome measure for the completing school domain 



Outcome measure 


Description 


Earned a diploma or GED certificate 


This binary outcome was measured by the Career Academies Four-Year Post-High School Follow-Up Survey administered four years after a student’s 
projected 12th-grade year. Students responded whether they had earned a high school diploma (on-time or late) or earned a GED certificate (as cited in 
Kemple, 2004). 
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Appendix A3.1 



Summary of study findings included in the rating for the staying in school domain 1 









Author’s findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 3 




Statistical 






Study 


Sample size 


Career Academies 


Comparison 


Career Academies 




significance 5 


Improvement 


Outcome measure 


sample 


(students) 


group 


group 


- comparison 


Effect size 4 


(at a = 0.05) 


index 6 



Kemple & Snipes, 2000 (randomized controlled trial) 7 



Dropped out of high school (%) High-risk youth 


345 


21.3 

(41.1) 


32.2 

(46.9) 


10.9 


0.34 


Statistically 

significant 


+13 


Domain average 8 for staying in school 










0.34 


Statistically 

significant 


+13 



1. This appendix reports findings for the high-risk youth that were considered for the effectiveness rating and the average improvement indices. Findings from the low- and medium-risk subgroups 
are not included in these ratings, but are reported in Appendix A4.1 . 

2. The standard deviation for students in each group shows how dispersed participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more 
similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this outcome. 

3. For this outcome, the mean difference was calculated so that a positive effect was found when fewer intervention youth than comparison youth dropped out of school (comparison group mean 
minus the intervention group mean). 

4. Effect sizes for dichotomous variables are computed using the Cox index. For further explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to 
calculate statistical significance. In the case of Kemple and Snipes (2000), no corrections for clustering or multiple comparisons within this outcome domain were made. 

8. This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A3.2 Summary of study findings included in the rating for the progressing in school domain 1 









Author’s findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 




Statistical 






Study 


Sample size 3 


Career Academies 


Comparison 


Career Academies 




significance 5 


Improvement 


Outcome measure 


sample 


(students) 


group 


group 


- comparison 


Effect size 4 


(at a = 0.05) 


index 6 



Kemple & Snipes, 2000 (randomized controlled trial) 7 



Total course credits earned 


High-risk youth 


316 


19.3 

(6.9) 


17.3 

(6.9) 


2.0 


0.29 


Statistically 

significant 


+11 


Credits earned met graduation 
requirements (%) 


High-risk youth 


316 


39.9 

(49.1) 


26.2 

(44.1) 


13.7 


0.38 


Statistically 

significant 


+15 


Domain average 8 for progressing in school 










0.33 


Statistically 

significant 


+13 



1. This appendix reports findings for the high-risk youth that were considered for the effectiveness rating and the average improvement indices. Findings from the low- and medium-risk subgroups 
are not included in these ratings, but are reported in Appendix A4.2. 

2. The standard deviation for students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had 
more similar outcomes. For the total course credits earned, the standard deviation was provided by the author and is not reported in Kemple and Snipes (2000). For the credits meet graduation 
requirements outcome, the standard deviation was derived by the WWC from a dichotomous variable. 

3. Sample size provided by the study author and differs slightly from those reported in Kemple and Snipes (2000). 

4. Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used 
to calculate statistical significance. In the case of Kemple and Snipes (2000), a correction for multiple comparisons was made, but the significance levels did not differ from those reported in the 
original study. 

8. This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A3.3 Summary of study findings included in the rating for the completing school domain 1 









Author’s findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 




Statistical 






Study 


Sample size 


Career Academies 


Comparison 


Career Academies 




significance 4 


Improvement 


Outcome measure 


sample 


(students) 


group 


group 


- comparison 


Effect size 3 


(at a = 0.05) 


index 5 



Kemple, 2004 (randomized controlled trial) 6 



Earned a diploma or GED High-risk youth 

certificate (%) 


360 


82.7 

(37.9) 


83.2 

(37.5) 


-0.5 


-0.02 


ns 


-0.1 


Domain average 7 for completing school 










-0.02 


ns 


-0.1 



ns = not statistically significant 

1. This appendix reports findings for the high-risk youth that were considered for the effectiveness rating and the average improvement indices. Findings from the low- and medium-risk subgroups 
are not included in these ratings, but are reported in Appendix A4.3. 

2. The standard deviation for students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had 
more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this outcome. 

3. Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations . 

4. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

5. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

6. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to 
calculate statistical significance. In the case of Kemple (2004), no corrections for clustering or multiple comparisons within this outcome domain were made. 

7. This row provides the study average, which in this case is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The 
domain improvement index is calculated from the average effect size. 
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Appendix A4.1 



Summary of low-risk and medium-risk subgroup findings for the staying in school domain 1 









Author’s findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 3 




Statistical 






Study 


Sample size 


Career Academies 


Comparison 


Career Academies 




significance 5 


Improvement 


Outcome measure 


sample 


(students) 


group 


group 


- comparison 


Effect size 4 


(at a = 0.05) 


index 6 



Kemple & Snipes, 2000 (randomized controlled trial) 7 



Dropped out of high school (%) 


Low-risk youth 


385 


1.9 

(13.7) 


2.9 

(16.8) 


1.0 


0.26 


ns 


+10 


Dropped out of high school (%) 


Medium-risk youth 


724 


9.0 

(28.7) 


8.0 

(27.2) 


-1.0 


-0.08 


ns 


-3 



ns = not statistically significant 

1. This appendix presents findings for the low-risk and medium-risk youth for measures that fall in the staying in school domain. Findings for youth who were at high risk for dropping out were used 
for rating purposes and are presented in Appendix A3.1. 

2. The standard deviation for students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had 
more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this outcome. 

3. For this outcome, the mean difference is calculated so that a positive effect is found when fewer intervention youth than comparison youth drop out of school (comparison group mean minus the 
intervention group mean). 

4. Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical 
Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple and Snipes (2000), no correction for clustering within this 
outcome domain was made. 
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Appendix A4.2 Summary of low-risk and medium-risk subgroup findings for the progressing in school domain 1 









Author’s findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 














Mean difference 




Statistical 






Study 


Sample size 


Career Academies 


Comparison 


Career Academies 




significance 4 


Improvement 


Outcome measure 


sample 


(students) 


group 


group 


- comparison 


Effect size 3 


(at a = 0.05) 


index 5 



Kemple & Snipes, 2000 (randomized controlled trial) 6 


Total course credits earned 


Low-risk youth 


376 


24.4 

(4.0) 


23.6 

(3.3) 


0.80 


0.22 


ns 


+9 


Total course credits earned 


Medium-risk 


687 


22.6 


22.9 


-0.30 


-0.06 


ns 


-2 




youth 




(5.1) 


(4.9) 










Credits earned met graduation 


Low-risk youth 


376 


85.7 


74.6 


11.1 


0.43 


Statistically 


+17 


requirements (%) 






(35.1) 


(43.7) 






significant 




Credits earned met graduation 


Medium-risk 


687 


64.8 


65.2 


-0.4 


-0.01 


ns 


-0.4 


requirements (%) 


youth 




(47.8) 


(47.7) 











ns = not statistically significant 

1. This appendix presents findings for the low-risk and medium-risk youth for measures that fall in the progressing in school domain. Findings for youth who were at high risk for dropping out were 
used for rating purposes and are presented in Appendix A3. 2. 

2. The standard deviation for students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had 
more similar outcomes. For the total course credits outcome, the standard deviations were provided by the study author and are not reported in Kemple and Snipes (2000). For the credits meet 
graduation requirements outcome, the standard deviation was derived by the WWC from a dichotomous variable. 

3. Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations . 

4. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

5. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

6. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical 
Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple and Snipes (2000), no correction for clustering within this 
outcome domain was made. 
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Appendix A4.3 Summary of low-risk and medium-risk subgroup findings for the completing school domain 1 









Author’s findings from the study 

Mean outcome 
(standard deviation 2 ) 




WWC calculations 










Career 




Mean difference 




Statistical 






Study 


Sample size 


Academies 


Comparison 


Career Academies 




significance 4 


Improvement 


Outcome measure 


sample 


(students) 


group 


group 


- comparison 


Effect size 3 


(at a = 0.05) 


index 9 



Kemple, 2004 (randomized controlled trial) 6 



Earned a diploma or GED certificate (%) 


Low-risk youth 


376 


100.0 

(0.0) 


99.4 

(7.7) 


0.6 


na 


ns 


na 


Earned a diploma or GED certificate (%) 


Medium-risk 

youth 


722 


92.9 

(25.7) 


92.1 

(27.0) 


0.8 


0.07 


ns 


+3 



na = not applicable 

ns = not statistically significant 

1. This appendix presents findings for the low-risk and medium-risk youth for measures that fall in the completing school domain. Findings for youth who were at high risk for dropping out were 
used for rating purposes and are presented in Appendix A3. 3. 

2. The standard deviation for students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had 
more similar outcomes. The standard deviation was derived by the WWC from a dichotomous variable for this measure. 

3. Effect sizes for dichotomous variables are computed using the Cox index. For an explanation of the effect size calculations, see 'echnical Details of WWC-Conducted Computations . An effect 
size cannot be calculated when there is no variance in outcomes among the groups. 

4. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

5. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. The improvement index cannot be estimated when an effect size cannot be 
calculated. 

6. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple 
comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical 
Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kemple (2004), no correction for clustering within this outcome 
domain was made. 
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Appendix A5.1 



Career Academies rating for the staying in school domain 1 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 2 

For the outcome domain of staying in school, the WWC rated Career Academies as having potentially positive effects. It did not meet the criteria for positive effects 
because it had only one study that met WWC standards. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative effects) 
were not considered because Career Academies was assigned a higher applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. 

Met. The only study on Career Academies reported a statistically significant and substantively important positive effect in this domain. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects or indeterminate effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Career Academies has only one study meeting WWC evidence standards. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 



1. This intervention rating was calculated based on 474 youth in the study sample who were most at risk of dropping out of high school. Researchers used student characteristics at baseline 
(including whether students had a sibling who dropped out, was overage for grade, had transferred schools two or more times, and their attendance, GPA, and credits earned in the year of 
random assignment) to develop a model to predict whether students in the comparison group dropped out of school and then applied theses predictions to the intervention group students. The 
full study sample included 1,764 youth. 

2. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings 
of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5.2 



Career Academies rating for the progressing in school domain 1 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 2 

For the outcome domain of progressing in school, the WWC rated Career Academies as having potentially positive effects. It did not meet the criteria for positive 
effects because it had only one study that met WWC standards. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, and negative 
effects) were not considered, because Career Academies was assigned a higher applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. 

Met. The only study on Career Academies reported a statistically significant and substantively important positive effect or indeterminate effects in 
this domain. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects or indeterminate effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Career Academies has only one study meeting WWC evidence standards. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 



1. This intervention rating was calculated based on 474 youth in the study sample who were most at risk of dropping out of high school. Researchers used student characteristics at baseline 
(including whether students had a sibling who dropped out, was overage for grade, had transferred schools two or more times, and their attendance, GPA, and credits earned in the year of 
random assignment) to develop a model to predict whether students in the comparison group dropped out of school and then applied theses predictions to the intervention group students. The 
full study sample included 1,764 youth. 

2. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effects. The WWC also considers the size of the domain level effects for ratings 
of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5.3 Career Academies rating for the completing school domain 1 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 2 

For the outcome domain of completing school, the WWC rated Career Academies as having no discernible effects. It did not meet the criteria for other ratings (posi- 
tive effects, potentially positive effects, mixed effects, potentially negative effects, and negative effects) because the one study that met WWC standards did not show 
statistically significant or substantively important effects. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1 : None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Met. The WWC analysis found no statistically significant or substantively important effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Career Academies has only one study meeting WWC evidence standards. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. 

Not met. The WWC analysis found no statistically significant or substantively important positive effects in this domain. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. The WWC analysis found no statistically significant or substantively important effects, either positive or negative, in this domain. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. The WWC analysis found no statistically significant or substantively important effects in this domain. 

(continued) 
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